Currently, requests for higher quality video are ever increasing. Video content demands tend to go to higher resolution, higher frame-rate, and higher bit-depth. To combat the bit-rate increase corresponding to high definition (HD) video and other bit rate intensive developments, especially to meet the transmission constraint of network and communication techniques, new technologies to further reduce bit-rate are strongly demanded.
There are at least two basic approaches to reducing compression bit rate. The first approach involves improving compression technology, and the second approach involves performing some sort of preprocessing prior to compression.
With respect to the first approach, namely improving compression technology, the progression of the same can be readily seen in the development of the various Moving Picture Experts Group video coding standards, such as, for example, the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) MPEG-1 Standard, the ISO/IEC MPEG-2 Standard, the ISO/IEC MPEG-4 Standard, and the ISO/IEC MPEG-4 Part 10 Advanced Video Coding (AVC) Standard/International Telecommunication Union, Telecommunication Sector (ITU-T) H.264 Recommendation (hereinafter the “MPEG-4 AVC Standard”).
For most video coding standards, increasing the quantization step size is a means used to reduce the bit-rate. However, this technique can result in severe blocky artifacts and other coding artifacts due to the loss of high frequency details.
With respect to the second approach, namely performing some sort of preprocessing prior to compression, the goal of such preprocessing is to remove the information that is least important in terms of visual perception, or the information that is able to be recovered after the decoding process without significantly altering the content. This bit rate reduction is also commonly referred to as data pruning. Some common techniques to perform this bit rate reduction through data pruning are the use of low-pass filters and down-sampling (which can be seen as a filtering process), followed by an up-sampling at the decoder. One effect of these schemes is that the decoded and reconstructed video looks a bit blurry since these techniques are designed to eliminate high frequency information in the video in order to reduce the bit-rate.
As for interpolation which, for example, can be used for the previously described upsampling, a wide range of interpolation methods and schemes have been discussed and developed, beginning with conventional bilinear and bi-cubic interpolation and continuing to sophisticated iterative interpolation methods such as projection onto convex sets (POCS) and non-convex nonlinear partial differential equations.
To avoid the jerkiness occurring along edges, edge-oriented interpolation methods using a Markov random field and covariance of the low resolution image have been proposed.
One prior art approach employs a combination of directional filtering and data fusion to estimate the missing high resolution (HR) pixels by a linear minimum mean square error (LMMSE). Another group of interpolation algorithms predict the fine structure of the FIR image from its corresponding low resolution (LR) image using different kinds of transforms such as the wavelet transform or the countourlet transform.
Each of the above methods are suitable for up-sampling the same ratio in both horizontal and vertical directions; that is, in a fixed and regular data grid (i.e., all data points are found in a rectangular grid). However, when interpolation is used along with data pruning, flexibility is desired in order to adapt to the discarded data and to adjust to the varying surroundings of each pixel to achieve the best performance.